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ABSTRACT 

This docuaent provides inforaation about the 
developaent of a quality control procedure that adds a nev capability 
to prograa aanagenent. The proceduret the Standardized Educational 
Prograa Audit (SEPA) , is being developed by the Florida Department of 
Education for use vith Blenentary Secondary Education Act Title III 
developaent projects. This report is intended to acquaint educational 
■anagers and technicians with this particular approach to the audit 
problem, vith the procedure that has been produced at this point in 
its development* with its effectiveness* and vith technical problems 
yet to be overcome. SEPA can help in the early detection of technical 
problems not only vithin development projects but also among them. 
Being able to detect and measure the incidence of problems common 
among projects vill help the department of education increase its 
responsiveness to the needs of district developers. The use of 
problem information by projects vill increase their effectiveness and 
efficiency. It is expected that once SEPA is fully developed and 
tested it vill be adaptable for quality control applications in a 
variety of other program operations. (Author/DV) 
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FOREWORD 



Developlnq marraqement inforniation and systematic decision- 
making capabilities are high priorities In Florida education 
today. New capabilities and applications are being developed 
in manaqement and quality control. The purpose of such de- 
velopment is three-fold: 

• To increase the responsiveness of educational 
programs and services to the needs of the 
individuals or groups being served. 

. To improve the effectiveness of educational 
programs and services In meeting human needs, 

• To enhance the ef<i1ciency with which resources 
are used in providing programs and services. 

Th.5 document provides formative Information about tne develop- 
ment of a quality control procedure which adds a new capability 
to program management. The procedure, the Standardized Educa- 
tional Program Audit (SEPA), Is being developed by the Educa- 
tional Innovations Section for use with ESEA, Title III devel- 
opment projects. SEPA offers great promise in the early detection 
of technical problems not only within development projects, but 
also among them. Beina able to detect and measure the incidence 
of proble.Tis common among projects will provide a oowerful new 
tool. By using problem incidence Information, the Department of 
Education will be able to Increase Its responsiveness to the 
needs of district developers. The use of problem information 
by projects wHl Increase their effectiveness and efficiency. 

Once fully developed and tested, SEPA should be adaptable for 
quality control applications In a variety of other program 
operations. 




Ralph D. Turlington 
Connlss loner of Education 
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PART I 



THE STANDARDIZED EDUCATIONAL PROGRAM AUDIT: WHAT AND WHY? 



Context of the Problem 

In contemporary education, the development project is a popular tool for 
problem solving. For example. Title III of the Elementary and Secondary 
Education Act provides developmental capital to local school districts 
to solve persistent educational problems in innovative ways. The nature 
and scope of development projects have qreat variability; however, there 
are aeneric decisions which must be made both during the course of proj- 
ect development and at its conclusion. These decisions are of qreat 
importance In the Quality control of developmental endeavors and in the 
sharinn of project- developed practices and products with the educational 
community. 

In projects with varying degrees of complexity ana sophistication, prob- 
lems in production, operation, and management often occur. If these 
problems are allowed to go uncorrected, their consequences may prevent 
the completion or success of the project mission. If this should occur, 
the educational cormiunity would be deprived of important information. 
Educators would not know whether the innovative idea being developed 
and tested was a workable and productive solution to the problem being 
addressed. Therefore, systematic management and Quality control of 
project functions becomes imperative. 

The responsibility for problem detection must be shared. Eacn project 
must include procedures for process and outcome evaluation within its 
management design. The execution of these procedures becomes a local 
responsibility. The state agency responsible for the general manage- 
ment of all such projects within the state must have a co-equal respon- 
sibility. A state agency is in a unioue position to collect and analyze 
aggregate information about project- related problems from which strategies 
for general Improvement of projects can be derived. The responsibility 
of the state agency is two-fold. First, the state agency must help to 
detect and correct problems in individual projects, and second, it must 
help discover problems common to groups of projects and then to devise 
means to resolve common problems and to prevent their occurrence in 
future projects, when possible. 



StJte of Problem Areas 

To discharge its responsi billies in the quality control of development 
projects, the Florida Department of Education's Office of Educational 
Innovations has completed its second phase in the development of a 
Standardized Educational Program Audit (bEPA). The conceptual design. 



results* application, and proposed modification are the substance of 
this technical report. 

The three fundamental problems addressed in the development of SEPA are: 

* How can problems internal to project production, operation, 
and managenient be detected through the use of a standard 
procedure when qreat diversity exists among the focus , 
sophistication, complexity, and development status of pro- 
jects? 

* How can problematic information collected from many projects 
be aggregated at the state level to deter j the incidence 
of common technical problems? 

* How does a state agency use Information showing common sets 
of problems to design and deliver corrective Intervention 

to projects? 



lll^ Relationship of the Standardized Educational Program Audit 
to O ther Quality Control runctions 



SEPA is not a singular, discrete quality control mechanism. It is one 
of four interrelated quality control functions. These four functions 
are evalu ation, monitoring , auditing , and validation . While these 
functions differ In when, how, and by whom they are applied, they are 
each designed to obtain and provide critical decision-making Information 
at key points In project planning, operation, and diffusion. A brief 
explanation of each function is Important in understanding both Its 
purpose and the procedural design of SEPA. 



Evaluation 

Evaluation is a set of Internal quality control procedures for a project. 
In projects, evaluation Is: 

* Used to determine the presence, natuie, extent, significance* 
and course of learner changes In relation to project Inter- 
ventl'^n (i.e., whether the learner-oriented problem was 
solved). 

* Used to make formative and summatlve decisions about the 
tools and processes of the intervention being developed 
and tested. 
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* Used ar a manaqement tool to check anc regulate production 
dnd operation activities. 

Evaluation tasks are perfotwd by the project staff, district evaluation 

services, and/or by a contractor. Information obtained is used by the 

project and shared with the state agency and other audiences. 



Monitoring 

Monitorim, as uspti by the state agency, is external to the project. It 
is proce«:s-(»rientcd and consists of two basic strategies. First is the 
export: review and negotiation of project plans. This strategy is used 
to dptt'wt and correct {)0tentidl faults in: basic design; planned activi- 
tie^M U't)dl compliance; and general commrml cat ions between the project 
and tho state agency. The second strategy is Informal on-site reviews 
of th»> projects performed by DoE orogram specialists and staff members 
^rom the Fducational Innovations Section. 



Aud itinf) 

Auditing is external to projects, formal in nature, and procesi -oriented. 
SFPA is used to df^tect problems in project production, operation, and 
manaqef'K'nt at the project level. At the state agency level, summarized 
<iud'* • indings jr-' used to diagnose broad areas of problems among projects. 
The dudi*: procedure Is conducted by management and program experts trained 
to servo as audit teams. Auditors must be independent of local agency and 
state anency affiliations. In conducting their on-site activities, auditors 
obsorve, verify, and report existing conditions within projects. Compiled 
r-eports are submitted to the state agency, which In turn, deliver-" them to 
local aiency docision-makers and project personnel. 



Val idatiqn 

Validation Is outcome or accomplishment oriented. It is used to verify 
and certify the success-effectiveness, efficiency, and portability of the 
practice and products developed by a p.oject. The validation procedure 
is conducted by management, measur&nfcnt, and program experts trained to 
serve as validators. \/al1dat1on teams are Independent of the project, 
local agency, state agency, and ESEA, Title III. Validation Is a guallty 
control procedure applied before project-developed practices and products 
aie released for state and national dlsseifl1nat1on/d1f fusion. 
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Purpose of the^ Tecnni cal Re port 



This technical report is intended to acquaint educational manaqers and 
technicians with a particular approach to the audit probiem, the proce- 
dure whic^ has heon pro.iuccd at this point in development, its effective- 
ness, ar)d tf't hiiit »1 DrobUnis yet to be overcome. 

Reactions and criticises by readers of this report are solicited. Responses 
or rvnuost^ tor addit'unal information may be sent to: 

Ur, Ray C. Foster 
Fducational Innovations Section 
rionda Department of Education 
Room ?00. WJS Building 
Tdllahdssec, Plorida 32304. 



to 



PART II 



BASIC DESIGN AND OPERATION OF THE 1974 STANDARDIZED 
EDUCATIONAL PROGRAM AUDIT 



The Audit Premise 



The premise of auditing is cne of verification. The history of auditing 
can be traced to the days of the Roman Empire when accounts of property 
and aebts werp read to independent listeners who verified the accuracy 
of the records; hence, the term "audit." The independent inspection of 
records or conditions for the purposes of verfication and reporting is 
common practice in fiscal operations. Only recently has the concept of 

independent inspection and verification been adapted for educational 
program applications. 

Auditino is not evalution in the traditional sense. The auditor's 
function is to detect and report discrepant conditions or problems—not 
to correct the problems when discovered. 

Prerequisite Conditions for a Standardized 
Educational Program Audit Procedure 

The design incorporated in this reported SEPA procedure is based on three 
fundamental assumptions: 

1. That there are common critical concerns for all develop- 
ment projects, regardless of variability in program focus, 
stage of development, complexity or sophistication In 
design; 

2. That these common elements are addressed by and incor- 
porated in all projects; and 

3. That these common elements can be reduced and defined 
into sets of ob<:ervable phenomena from which standard 
audit items can be generated. 

Development projects engaged in educational problem-solving (managed by 
the Educational Innovations Section of the Florida Department of Educa- 
tion) all have standardizeo sets of planning and management elements. 
Each proj.JCt design includes: specific learner need statements; cri- 
teria for need resolution; a defined problem; a mission statement; 
terminal performance specifications lor learner change; program com- 
ponent and product specifications; a project manaqement schedule; a 
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pro.iet t fiandqement loq; and evaluation procedures with common data targets 
(e.o., t earner nerfornanLe change, sionificance of change, cause 
of chan«u?» functiund] utility measure».:ents of program components and 
product*;). Those cornon elerients for project design and management are 
standardised f.r oporation in Florida's Manual for Applicants and Grantees 
(Florida Uepa' t;>'nt of Education, 19>4). better known as the ESEA, TitTe"" 
III PACE Marv.ai. The .inpprtance of„thi s standardization in format and 
cpji_tent .annot be xirnirizeci. Uitjiout suc h ^standardization In de sign and 
P_105/*'^'"'''v./^'»'»"i' .<"An"ot^ ho d SEPA nrpcedure. The' re-design of Fforlda's 
project nrtn.jM, , ont system in 19>? was a" prerequ i s i te to the current 
audit dosi'ifi. 

A diaar.vn of the fonnat and content of project plans is presented as 
DIAGRAM I: u^'^ini national Fortnat for Grant Applications on the 
followinq immi-. The diaqr.iM is taken from the 1974 PACE Manual. 



B«^^VC-P,PA^9 '^ 0^ the 1974 SEPA Protocols : 
jj^A^/K^lg^tation and Procedure 



Instrument Content a nd Design 

The connon clenents and conditions expected to be found in all projects 
nanaqed throunh the Educational Innovations Section provided a basis 
for the qenerat^on of specific audit items. First, five areas were 
identified as inoortant areas in which technical problems could occur 
in pro.ierts. These areas were: Project Management (General); Production 
Management; MfMsuranent of Change; Product Development; and Program 
Operation. Then, specific itews in each of the five areas were" constructed 
to provide a rpr^ns of observing specific conditions in projects. 



Instrunert ForiMt Pes i qn 

After deterrinina the content elements for the audit instrument, the 
next steps were the formatting of items and the designing of procedures 
for iten use by auditors. Two approaches to item formatting were used 
in the instrument design. 

The first iton format was the "expected versus observed conditions com- 
oarison format." Thirty-three (33) of the thirty-seven (37) items in 
the five sections of the instrument used this format. In this format, 
the auditor was given an expActed condition with which he then compared 
the conditions ne observed on-site. If the observed and expected con- 
ditions matched, then a "consistency" was reported. If the expected 
condition was partially— but not completely— fulfilled, then a "dis- 
crepancy" was reported and explained. If an applicable, expected con- 
dition was not found to exist, then an "omission" was reported. When 
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an expected condition was determined not applicable to a project, then 
the non-applicability was reported and explained. The auditor occasionally reported 
additional observations" related to the expected conditions. Additional 
observations reported or explained conditions, rather than the auditor's 
judgment or opinion on the matter. 

The "expected versus observed condition comparison format" was used to 
standardize response modes for the thirty-three (33) items using this 
format. Standardized response modes provided the capability of summa- 
rizinq responses across projects on given items or item groupings. This 
format also was useful in reducing auditor subjectivity in applyij'^j items 
and in reporting conditions as observed. 

The procedure used by auditors in applyinq items In the "expected versus 
observed condition comparison format" is illustrated in DIAGRAM 2: Audit 
Observation and Reporting Procedure on the following page. 

In order to simplify the on-site recording of data for auditors, an 
instrument form was devised which would allow the auditor to check the 
appropriate status finding for an expected condition by a given audit 
item. Adjacent space was provided in which the auditor could record 
observations and notes for later use in the preparation the audit 
report. The instrument form was printed on Bh" by 14" paper to pro- 
vide space for such recording. A reduced example of a page from the 
instrument form is provided in Exhibit 1. Instrument Form Example Page 
on page ten. 

The other item format used in the audit instrument was a narrative or 
constructed response format. In this format, the auditor was given a 
statenient or question to which he was to respond. While such responses 
tend to elicit more subjective responses, it was thought that constructed 
responses to general questions might provide a source of useful infor- 
mation that was broader in scope than the type of information obtained 
in the first format. Four items-, two in Section V-B and two in Section 
V-C. used a constructed response format. A summary compliance question 
at the end of the audit report also used the constructed response for- 
mat. 



I nventory of Audit Items from 
the 1974 SEPA Instrument 



Presented below is the set of audit items used in the 1974 SEPA instru- 
ment. The items are presented under their respective area headings and 
are listed in the order of presentation in the field instrument. The 
items are used in Part III of this report for the presentation of re- 
sults. However, the items are described by key-word subjects in Part 
III to save space. 
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Section One: Project Management 

1. The project director can produce a project management schedule 
which specifies: 

a. the expected dates on which major production, operation, 
and management activities are to be completed; and 

b. the projected flow of the occurrence of activities over 
time and in sequential relationship. 

2. The project director can produce a project management log which: 

a. describes and documents major project events as they 
occur in chronological order; 

b. records the actual dates on which major production, 
operation, and management activities were initiated 
and completed (internal events) and the dates on which 
major external events affecting the nature or direction 
of project activities occurred; 

c. records decision alternatives for major decisions made 
in response to events and the actual decision made 
from the alternatives; and 

d. describes the rationale for any significant departure 
from planned activities, established completion dates, 
and/or flow patterns. 

3. The project director can produce evidence that progress (status) 
information relative to scheduled activities has been reviewed 
formally by the project staff on a recurring cycle. 

4. The project director can produce evidence that progress (status) 
information relative to scheduled activities has been reviewed 
formally by LEA decision-makers at the district level on a 
recurring cycle. 

5. The project director can produce evidence that progress (status) 
information relative to scheduled activities has been reviewed 
formally by a project advisory committee on a recurring cycle. 

6. The project director can produce evidence that progress (status) 
information relative to scheduled activities has been or will be 
reviewed periodically by other audiences (e.g., other participa- 
ting agencies, parents of the beneficiaries, DoE program spe- 
cialists, etc.) during the project grant phase. 

7. The project director can provide a written plan which estab- 
lishes the current and future provisions for the continuation 
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of project endeavors (contingent upon established effectiveness) 
after federal funds are withdrawn. 



Section Two : Production Management 

1. Each product to be produced during the current grant phase has: 

a. a specified function for which It Is to be used when 

complete; 

b. a set of requirements to be met In production; and 

c. a set of standards which must be met when the finished 
product is to be used. 

2. For each product, the production activities (or tasks) and their 
associated milestone dates are included in the project manage- 
ment schedule. 

3. The project director can provide evidence that the current 
production status for each of the products being prepared 
during this grant phase is within two weeks of its scheduled 
production status as set forth in the project management 
schedule. (This applies to a delay, not to an advance.) 

4. If any delay greater than two weeks exists between the actual 
production status and the expected (scheduled) production 
status of a product, then that delay is recorded in the pro- 
ject management log. 

5. For each product being developed for Intervention with the 
target population, there is a specified evaluation procedure 
to assess: 

a. the completeness of the product in relationship to 
the specified criteria or standards for the product; 
and 

b. the functional utility of the product In relationship 
to the specified criteria or standards for the product. 

6. The evaluation procedure(s) designed to assess the completeness 
and functional utility of each product is (are) being fully 
implemented and is (are) being conducted on schedule. 

7. The evaluation findings for each product are being fully 
documented as those findings are determined. 
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8. The project director can provide evidence that the functional 
util it y of any and all products (designed for intervention use 
with learners) has been demonstrated through appropriate evalua- 
tion procedures before being disseminated to persons, agencies, 
or institutions other than those participating in the develop- 
ment or testing of said products. 

9. All learner intervention products still In formative stages of 
development are marked to indicate that their functional utility 
has not been demonstrated through appropriate evaluation proce- 
dures and are thereby subject to extensive revision or elim- 
ination from intervention usage. 



Section Three: Measurement of Change 

1. E. :h terminal performance specification (learner change objective) 
contains the necessary information to render it measurable at a 
given point in time (i.e., the specifications include: (1) what 
portion of the target population will be able to do what; (2) at 
what level of performance; and (3) under what conditions). 

2. Each terminal performance specification (learner change objective) 
has an identifiable evaluation design which when conducted will 
provide the information necessary to determine whether the ex- 
pected portion of the target population demonstrated the expected 
performance at the level(s) specified (i.e., the actual attainment 
of the objective). 

3. Each terminal perfornjajic^ specification (learner change objective) 
is provided with evaluation procedures which will yield information 
for: '» ■ 

a. determining the statistical significance of learner 
change; and 

b. rendering conclusions about the nature and cause of 
learner change in relationship to the program of 
intervention. 

4. Each terminal performance specification (learner change objective) 
has identifiable and documented data collection procedures for 
implementing the evaluation design(s) reviewed in Items 2 and 3 
above. 

5. If samoling procedures were used to structure the data collection 
process for the summative evalution of a terminal performance 
specification (learner change objective): 

a. the rationale (for sample size, selection, assignment, and 
assumed equivalency between comparison groups) is presented 
in project documents; and. 
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b. there is no evidence to suggest that the statistical assump- 
tions established for the sampling procedure(s) have been vio- 
lated in Implementation. 

The instrument (s) (whether coranerclelly or locally developed) used 
in the collection of data for the measurement of learner change 
for each terminal performance specification (learner change objec- 
tive) is (are): 

a. appropriate for the target population; 

b. sensitive to measure the range and scope of the behavior of 
concern; 

c. has (or will have) an established and documented estimate of 
Its (their) reliability; and 

d. can be judged as valid for measuring the behavlor(s) to be 
measured. 

For each terminal performance specification (learner change objec- 
tive) undergoing summatlve evaluation, there is evidence Indicating 
that the personnel responsible for implementing the data collection 
procedures are competent (certified, qualified, and/or specifically 
trained through inservlce activities) to conduct the data collection 
tasks to which they are assigned. 

For each terminal performance specification (learner change objec- 
tive) undergoing summatlve evaluation, there is evidence that the 
data collection and processing procedures are being managed in an 
accurate manner at key points in collection and processing. 

For each learner performance specification (learner change objec- 
tive) there are Identifiable and documented data analysis procedures 
being used to Implement its evaluation design. The data analysis 
procedures when fully implemented will yield information for deter- 
mining: 

a. whether the expected portion of the target copulation demon- 
strated the expected behavior(s) at the level(s) specified; 

b. whether the measured change In learner behavior has statistical 
significance; and 

c. whether the nature and cause of change In learner behavior is 
attributable to program intervention. 

All suiiinative evaluation procedures and tasks are being Implemented 
in accordance with the approved grant application and project 
management schedule. 
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Section Four; Product Development 

1. Based upon a review of each program intervention product devel- 
oped to date there is evidence to suggest that each product Is matched 
directly to one or more of the specific learner needs and problems 
Identified in the approved grant application. 

Z. Based uoon a rpview of each program intervention product developed 
to date, there is no evidence to suggest that all requirements 
established for each product In the approved grant application 
have failed to be met. 

3. Based upon a review of each program intervention product devel- 
oped to date, there is no evidence to suggest that all functional 
utility standards established for each product In the approved 
grant application have failed to be met. 



Section Five: Program Operation (A ) 

1. There is evidence to show that the number and type of direct 
participants recorded In the approved grant application are 
participating in program activities. 

2. There is evidence to show that the qualifications of personnel 
paid from ESEA, Title III funds match the personnel specifications 
set forth in the approved grant application. 

3. There is no evidence to suggest that the project personnel paid 
with ESEA, Title III funds are being used by the LEA for purposes 
other than the accomplishment of the project mission and its 
associated functions. 

4. There is evidence to show that provisions were made for students 
and/or personnel from non-profit private schools to participate 
in program activities (correspondence, minutes, records, etc.). 



Section Five; Program Operation (B ) 

1. Identify and describe the organizational polnt(s) in the classroom, 
school, and/or district organization at which project Intervention 
occurs. 

2. Identify and describe the basic unit(s) in which Intervention is 
delivered. 
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Section Five: Program Operation (C) 

(Prepare a narrative anst^er to the two following questions) 

1. Do all program components Interrelate one with another, demon- 
strating both uniformity of fit and Integrity as a whole? 

2. To what extent does the program of intervention, as a comprehensive 
solution, afford coverage for the learner needs and problems ad- 
dressed by the project? 



Operational Procedures Used to Conduct 
the S£pA MUs^on far IW 



The fol lowing sub-sections provide the reader with a general overview 
of the processes involved in conducting the SEPA mission for development 
projects In 1974. 



Project Preparation 

Thirty (30) ESEA, Title III development projects were audited In early 
1974. Each project manager and his district superintendent were noti- 
fied eight weeks prior to the audit and given pertinent Information about 
it. Project managers prepared "pre-audit Information packages" for the 
audit team to be assigned to the project. The pre-audit Information 
package contained the following Items: 

1. A key personnel inventory for the project. Including members 
of the project advisory committee and project beneficiaries. 

2. The current approved grant application with any addenda or 
amendments . 

3. Quarterly reports for the current budget period. 

4. The project management schedule. 

5. The project management log entries from the first day of the 
current budget period to the date of request. 

6. Results of evaluation procedures conducted during the current 
grant period. 

7. Evaluation and audit reports from the preceding budget period. 

8. Evaluation Information forms for each terminal performance 
specification (learner change objective). 
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9. Comoleted product summary sheets for all intervention products 
completed by the date of request. 

All pre-audit information packages were prepared and submitted to the 
Educational Innovations Section prior to the auditor training session. 



Contr ac t Preparation 

In order to expedite the recruitment of auditors and the processing of 
paperwork, a contract (DoE #740-091) was prepared and executed between 
the Department of Education and the Florida Educational Research and 
Development Council, Inc., a private, non-profit educational corpora- 
tion. The contractor handled all matters relating to the recruitment 
of auditor candidates, contracting with auditors, payment of honoraria 
and travel expenses. The Educational Innovations Section was the sole 
judge for candidate selection, acceptability of work, and authorization 
of payment to auditors. 



Selection/Training/Assiqnment of Auditors 

Auditor candidates were submitted by the contractor for review and selec- 
tion. Two auditors were selected for each of the thirty (30) projects 
to be audited. One member of the audit tedrn was required to possess 
triiining and experience in management and evaluation. The other team 
mefiiber was required to possess training and experience in the focus 
area of the assigned project. All auditors selected came from either 
school districts or universities. Fifty-three (53) of the sixty (60) 
auditors selected were holders of doctorates. Twenty-one (21) of the 
management auditors had been auditors of projects in 1973. 



A one-day training session was required of selected candidates to 
serve as an auditor. The purpose of the training session was to orient 
the "trainees" to the audit procedures and to allow audit team members 
to become acquainted. Instruction was provided in observation/reporting 
procedures on an item by item basis to auditors in their speciality 
croups. Overall reporting procedures were explained and time lines pro- 
vided. The following subjects were covered during the course of the train- 
inn session: 

1. Project planning and managenient designs contained in the new 
PACE Manual. 

2. The audit as one of four quality control functions. 

3. The SEPA Mission for 1974. 

4. General responsibilities of the auditor. 

5. Auditor decorum on-site. 
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6. Channi>>. of coitinunicatlon 

7. Pre-audit preparation tasks 

8. The SEPA design 

9. Audit team responsibilities 

10. SEPA Report Drafts 

11. SEPA Report Elements 

12. Basic rules for report preparation 

The pre-audit Information packages submitted previously by project mana- 
gers were distributed to audit teams. Planning time for autiit teams was 
provided at the close of the training session. 



Auditor assignments to teams and projects were based on the following 
factors: 

1. appropriate speciality 

2. independence of local district or project affiliation 

3. accessibility to the project site 

4. special factors. 

Special factors pertained to unique aspects of the project being audited. 
As an example, for a bi -lingual education project in Dade County, both 
auditors assigned were speakers of Spanish. The on-site audit was con- 
ducted in Spanish and reported in English. 



Pre-Audit Activities 

Following the training session, audit teams reviewed the pre-audit 
information packages and prepared their audit questions, needs for 
additional information, and interviews to be conducted during the on- 
site phase. Audit teams were charged with the responsibility of schedu- 
ling the on-site audit activities at a time mutually convenient for 
both the audit team and the project staff. The audit was to be accom- 
plished during the month of February on any two consecutive weekdays. 
A calendar of audit events for 1974 is provided on page nineteen. 
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AUDIT EVENT 



DATE 



1. Project Managers and their respective 
district Superintendents notified of 
1974 SEPA Plans. 

2. Contractor submits list of auditor candi- 
dates to Educational Innovations Section 
(EIS). 

3. EIS submits list of selected auditor candi- 
dates to contractor. 

4. Auditor candidates notified of Training 
Session. 

5. All Pre-Audit Information Packages received 
from Project Managers. 

6. Auditor Training Session conducted. 

7. On-site auditing period begins. 

8. On-site auditing period ends. 

9. All audit report drafts submitted by audit 
teams to Educational Innovations Section. 

10. Report delivery schedule established 
with Project Managers and Superintendents. 

11. Benin report delivery schedule established 
with Project Managers and Superintendents. 

12. Complete report delivery phase. 

13. Beain technical report preparation. 



12/14/73 

12/21/73 

12/21/73 

12/21/73 

01/18/74 

01/23/74 
01/30/74 
03/01/74 
03/08/74 

03/15/74 

04/01/74 

04/30/74 
05/01/74 



On -Site Activities 

During the two-day period of on-site audit time, the audit team members 
were required to complete the thirty-eight (38) audit observations 
contained in the SEPA protocols. During this phase, evidence was exam- 
ined and interviews were conducted in order to determine and verify 
existing conditions. When completed, the audit team exited: no 
exit interview was conducted. The only permitted reporting of audit 
findinas was the official SEPA report, returned to district and project 
personnel by members of the Educational Innovations Staff. 
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!^.^PPrA Prepdrati on by AudU Teams 

Followinq the on-site activities each audit team compiled its find- 
ings into a report draft and submitted it to the Educational Inno- 
vations Section. Each of the thirty (30) reports were prepared in 
the followinc^ format: 

1. Cover page with general project and auditor information; 

2. Auditor assurances page including signed statements for 
report accuracy and project independence; 

3. Audit activities description; 

4. Audit inventory. Sections I-V; 

5. Summary of Detected Problems; 

6. Summary Opinion of Project Compliance with the terms and 
conditions of its grant application; and 

7. Supporting information including a contact record of persons 
.nterviewed and the vitae of the auditors. 

Upon receipt of each report draft, the Educational Innovations Section 
Staff reviewed the document for completeness, accuracy, internal consis- 
tency, and propriety of content. Errors in reports were corrected with 
auditors via telephone. Statements of opinion presented in the Audit 
Inventory, Sections I-V were removed. When all corrections were made, 
the report was typed in final manuscript form and reproduced for delivery. 



Delivery of Repor ts to District/Project 

While final reports were being prepared, a report delivery schedule was 
established. A two hour meeting was scheduled for each audit report 
in the district where the project was located. The district superintendent 
director of federal programs, program directors, and the project manager 
and project staff were requested to attend the reporting session. 



A staff member from the Educational Innovations Section conducted the 
reporting session. During the session, the SEPA concept was explained, 
report findings were reviewed, detected problems were discussed, and 
joint measures for problem correction were established. 
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PART III 

FIELDTEST OF THE 1974 STANDARDIZED EDUCATIONAL PROGRAM AUDIT: 
SOME FINDINGS, CONCIUSTIONS, AND RECOf«^ENDATIONS 



Plan for the Format jve Evaluation 
of the 1974 SEPA Procedure 

The formative evaluation plan for the 1974 SEPA was designed to answer 
three basic questions: 

1. Did the application of SEPA result In the detection of 
technical problems In Individual projects? 

2. Could the &ata collected from individual projects be 
aggregated in a form which would permit measurement 
of the incidence of technical problems among projects? 

3. Based on the overall results of SEPA, what portions of the 
instrument and procedure should be revised for 1975? 

In order to answer these questions, data from SEPA reports were col- 
lected and analyzed in a variety of ways. 



For Individual projects, performance data were collected and presented 
in summarized tabular form. This tabulation permitted an overall review 
of individual project performance across thirty-three (33) objective 
audit items as well as a review of the performance of thirty (30) 
projects on a given item. From this summary, percentage scores for 
positive performance were determined for each project. The percentage 
scores were cast into Intervals to produce a graphic frequency iistri- 
bution of performance scores. The distribution permitted a viev of 
general project performance across the thirty-three (33) objective 
audit items. This information then was used to answer question #1 
above. 



To answer question #2, two tasks were performed: (1) the cons true ti j . 
of a "oroblem incidence profile" which yielded the rate of occurrence 
of technical problems among audited projects, and (2) the construction 
of a matrix which deplete'* the performance of project groups on the 
five sections of the audit Inventory. The "project group by section 
performance matrix" reported in each of Its cells: 

a. the rate of reported consistencies; 

b. the combined rate uf reported discrepancier. and omissions; and 

c. the rate of Item non-applicability per section per project 
group. 
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The rates of section non-applicability for project groups was Included 
with the other information to answer question #3, 



Analyses of responses to the five narrative items from the audit inventory 
were performed also. The five narrative items were examined in two ways: 
for the content of response and for the quality of response. This pro- 
vided additional information for questions #2 and #3. 



The results of the formative evaluation of the 1974 SEPA procedure are 
reported below in this section. The reader is advised to consider the 
findings presented herein as formative, rather than conclusive, in nature. 



Fieldtest Data Collection 

The data presented in this section were collected by two-member audit 
teams on two consecutive days in the month of February, 1974. Thirty 
(30) ESEA, Title III projects were audited. Projects audited were in 
one of five funding phases. Three projects (0-1-A) had been operational 
for approximately six weeks when the audits occurred. Nine (9) projects 
(0-1-B) had been operational for approximately eight months when the 
audits were conducted. Eight (8) projects (0-2) were in their second 
operational phase and seven (7) projects (0-3) were in their third 
operational phase. Three (3) projects (DM-1), validated in 1973, were 
operating as demonstration/diffusion centers, the first year for such 
projects. 



Problem Detection Within Projects 
Audit Findings for Projects 

The audit findings obtained from the thirty (30) projects 'audited 
1974 are presented in Table 1. This table provides the findings reported 
by the audit teams for the thirty-three (33) objective (expected 
versus observed condition) items found in the five sections of the 
audit inventory. 



Table 1 provides a status report (consistency - C, discrepancy - D, 
omission - 0, not-applicable - NA, or auditor deletion - X) for each 
of the thirty-three (33) items and for each of the thirty projects. 
(The deletion of an item by an auditor was performed when the response 
mode for an item did not include an NA response alternative. Therefore, 
such a deletion of an item is a report of item non-applicability.) 
Two types of summary information are provided for individual projects 
at the bottom of Table 1. The first summary gives the number of consis- 
tencies, discrepancies, omissions, not applicables, and deletions 
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for each of the audited projects. By subtracting the number of dis- 
crepancies and omissions from thirty-three and then dividing the dif- 
ference by thirty-three* a percentage score was obtained. The second 
summary (bottom line of Table 1) provides a percentage score for 
each project. 



As can be seen in Table 1, the ranges for status reports for projects 
were as follows: 



*range of consistencies reported: 

*ranqe of discrepancies reported: 

*range of omissions reported: 

*range for not applicable or 
auditor deleted items: 



6-33 per project; 
0-17 per project; 
0-13 per project; and 

0-10 per project. 



The percentage or "proficiency" scores for projects ranged from 18% 
to 100'^ with a mean score of 72.3% for the thirty projects audited. 
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Flqure 1 provides a frequency distribution of project scores in 
percentage intervals. One project scored in the 0-20% interval; one 
project in the 21-40% interval; five projects in the 41-60% Interval; 
eleven projects in the 61-80% interval; and twelve projects in the 
81-100?^ interval. The mean percentage score was 72.3%, thus Indicating 
that half of the projects audited were operating at or above the 72% 
proficiency level. 



FIGURE 1 

DISTRIBUTION OF AUDIT SCORES FOR PROJECTS 
(NUMBER OF PROJECTS « 30) 




0- ! 



0-20% 21-40% 41-60% 61-80% 81-100% 



Project Scores in Percentage Intervals 
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Conside rations i n Data Interpretation 

The interpretatiun of audit findings merits additional discussion. 



When an auditor marks a discrepancy for an audit Item, it indicates 
that a discrepant condition exists. The marking of a discrepancy does 
not indicate naqnitude or extent. For this reason, the auditor must 
report also the conditions observed when reporting a discrepancy. 



Differences in degree or extent may occur in discrepancies reported 
for two different projects on the same audit item. For example, 
a project with thirty (30) learner change objectives could have a 
discrepancy reported for missing a step in a data verification process 
for one objective. Another project having only two learner change 
objectives with multiple problems in data verification on both ob- 
jectives likewise would have a discrepancy reported. The extent of 
the discrepancy in the second project is relatively greater than the 
extent of the discrepancy in the first project. Although both proj- 
ects have discrepancies in kind, they differ in extent. Thus, dis- 
crepancies are "project-specific" in degree of occurrence. 



Just as there are differences in the extent of discrepancies between 
projects on a common item, there are differences in the magnitude 
of potential consequence in discrepancies on different Items for 
the same project. A discrepancy found In sampling procedures In 
a project's evaluation may be of greater magnitude of consequence 
than a reported error in management log documentation. Thus dis- 
crepancies are "item-specific" In magnitude of potential consequence 
on the project mission. 



In summary, discrepancies are "project-specific" In extent of occurrence 
and "iter' specific" in magnitude of potential consequence. Discrepancies 
can be sunined within and among projects In kind, but not degree or 
magnitude. 



Sunmarlzatlon and Conclusions 

The audit reports provided a unique source of independent information 
about projects. In discussing audit findings with the managers of the 
Individual projects audited, it was found that the problems, once re- 
ported, then were recognized and accepted by the project managers with 
few exceptions. The scoring procedures applied to project data enabled 
the Educational Innovations Section to easily locate projects with 
extreme deficits in performance. Individual identification of such 
projects was the first step In providing individualized corrective inter- 
vention. o>| 
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On the basis of (1) the nature and amount of information collected 
about the performance of individual projects* (2) the reaction of 
project managers in receiving audit information, and (3) the facility 
with which the state agency could locate and intervene with projects 
having extreme reported deficits, the SEPA procedure is considered 
an effective means of detecting technical problems within projects. 



Problems Encountered Within the SEPA Procedure 

Two main problems were encountered in the 1974 SEPA procedure. The 
first problem pertained to the auditor's collection and consideration 
of evidence from which status reports for items were rendered. Four 
project managers reported the occurrence of errors on from one to 
three Items in the set of thirty-three (33) objective items. Three 
auditor errors were responsible for this problem: 

(1) The auditor failed to request information which was 
available and appropriate to satisfy an Item expectancy; 

(2) The auditor failed to accept provided and appropriate 
evidence which would satisfy an item expectancy; or 

(3) The auditor ignored or overlooked information which 
authorized a change in project activity, and reported 
the authorized change as a discrepancy. 

Obviously, these auditor errors Impair the reliability of reported 
data. However, the Incidence rate of auditor errors of this nature 
reported by project managers was low (i.e., project manaoers from 
four projects reported such errors for one to three items per project). 



There is one other known auditor error that should not go unmentioned 
Occasionally, an auditor (especially a new auditor) Is unsure about 
reporting a discrepancy or a consistency for an item. This usually 
occurs when there is marginal evidence of a consistency or a minor 
evidence of a discrepancy. The error that occurs is a "benefit of 
the doubt" error In which a consistency is reported without appropriate 
accompanying explanation. The "benefit of the doubt" error reduces 
the sensitivity of the audit procedure as a problem detection mechanism. 
The Incidence rate for this type of error is unknown. One may spec- 
ulate that marginal discrepancies are being reported as consistencies 
since most reported discrepancies arc not marginal in nature. 



In order to improve the efficiency and effectiveness of the SEPA 
procedure, auditor errors must be reduced and maintained at the lowest 
possible level. To accomplish this, auditor training In the future 
will Include specific training and practice on (1) soliciting and judging 
evidence, and (2) decision-making and reporting in marginal situations. 
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Problem Detection Among Projects 



Aggregate Audit Findings (Objective Audit Items ) 

The audit findings obtained from the thirty audited projects were 
aggregated and analyzed in two different ways. The first aggregate 
analysis was performed to construct a "problem incidence profile" 
of detected problems among projects. The second aggregate analysis 
was performed to construct a "problem incidence profile" of detected 
problems among projects. The second aggregate analysis was performed 
to examine the performance of projects in different funding status 
groups on the five sections of the audit inventory. 



Table 2, "The Problem Incidence Profile" provides a summary of 
project performance adjusted for the applicability of items to projects. 
Column A orovides a listing of the thirty-three (33) objective items 
from the five sections of the audit inventory of the audit protocols. 
Column B gives the number and percent of projects in which a given 
item was applicable. The range of item applicability was from thirty 
projects or lOOC^ to eighteen projects or 60% for Item Five of Section 
III which pertained to sampling procedures in learner performance eval- 
uation. (Only 60% of the thirty projects used sampling procedures 
in their evaluation designs). Column C provides the number and percent 
of projects for which a consistency was reported. The consistency rates 
for items ranged from 100% to 23?^. The 23*^^ consistency rate was found 
for Item Seven, Section I pertaining to a contingency plan. Column D 
provides the number and percent of projects for which a discrepancy 
or omission was reported for each of the thirty-three items. The 
range of discrepancy rates found were from 0% to 77!^, the complement 
of rate*i for Column C. Column C illustrates the relative discrepancy 
rates for each of the thirty-three (33) objective items, thus creating 
a profile of incidence rates of technical problems reported for the 
thirty (30) audited projects. 



The Problem Incidence Profile reveals thr incidence rates of common 
problems among projects. A 35% criterion can be applied to the 
profile to identify those problems conmon to approximately one- 
third or more of the projects audited. When applied, this criterion 
identifies seventeen items or problems in four of the five audit 
inventory sections on which 35% or more of the projects had reported 
deficiencies. These items are listed below by audit inventory sections. 
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LISTING OF PROBLEMS WITH REPORTED 
INCIDENCE RATES OF 35% OR GREATER 



s£CTtON 




iU&JEtt t)litftEt>At<CV hAlE 


I 


2 


Mana()einent Log 


50%* 


PROJECT 






MANAGEMENT 


5 


CoiTBnunl cat ion- 


37% 






Advisory Committee 






7 


Continuation Contingency 


77%* 






Plan 




11 


1 


Product Specifications 


35%* 


PRODUCTION 






MANAGEMENT 


3 


Production Status 


47%* 




4 


Delay Documentation 


40%* 




5 


Product Evaluation Plan 


45%* 




6 


Evaluation Implementation 


44%* 


III 


3 


Causal -Comoarative Desian 


40% 


MEASUREMENT 






OF CHANGE 


5 


Sampling Procedures 


39% 




6 


Instrumentation 


40% 




8 


Data Accuracy Verification 


43% 




9 


Analysis Procedures 


59% 




10 


Implementation Status 


38% 


IV 


2 


Product Completeness 


38%* 


PRODUCT 






DEVELOPMENT 


3 


Product Utilit.v 


38%* 



The incidence rate for the problems reported on nine of the above items 
can be attributed to a change in c^uidelines. The 1974 SEPA proceduia 
was designed to correspond with the new guidelines presented in the 1974 
revision of Florida's PACE Manual. Many on-going projects, caught in 
the transition tn new guidelines, had not implemented completely the 
newly required changes at the time the audit was conducted. Those items 
so affected are marked with an asteri^Jt (*) in the above listing. 
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It is anticipated that the Incidence rate for these items will be reduced 
significantly in the 1975 SEPA findings. 



The problems with high incidence rates reported for the Measurement of 
Change Section of the audit inventory are of the greatest concern in the 
management of projects. These problems are associated with the production 
of credible evidence of the effectiveness of projects in solving the 
various problems they are addressing. Without credible evidence of 
success, projects cannot be "validated" nor subsequently shared with 
the educational community without engaging in hucksterism. Resolution 
of these problems is one of the utmost importance .to both the project 
and the state agency. Awareness of the existence and prevalence of the 
problem is the first step in designing and delivering corrective inter- 
vention. 



One other aggregate analysis was performed using the thirty-three (33) 
objective items. Aggregated data were analyzed in a two-way matrix in 
order to examine the objective items in the five sections of the audit 
inventory. Three types of information were placed in the twenty-five 
cells of the matrix: 

(1) the rate of reported consistencies per cell; 

(2) the rate of discrepancies, plus omissions per cell; and 

(3) the rate of non-applicable items per cell. 
The data are presented in Table 3. 



The purpose of this analysis was two-fold. The first purpose was to 
locate high discrepancy/ omission rates for projects in the different 
operation phase groups. This Information allows the selection of 
specific project groups for which to provide special corrective inter- 
vention of a basis of an operational phase commonality. The three 
greatest cell discrepancy/omission rates were as follows: 

(1) 0-2 projects for Project Management, 50%; 

(2) 0-2 projects for Production Management, 47%; and 

(3) 0-1-B projects for Mea;urement of Changes, 42%. 

These two project groups (0-2 and 0-1-B) each contained three projects 
with extremely discrepant scores in relationship to other projects in 
that group. These extreme scores increased the mean discrepancy rates 
for each group on the aforementioned audit inventory sections. Each of 
the three projects in the two groups had unusual circumstances surrounding 
them. In four of the six projects, the project managers had been changed 
in the course of the past year. The p^Jgjble relationship between the 
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replacement of project managers and the frequency of audit-detected 
problems tends to emphasize the central importance of the project 
manager and the requisite skills for the position. 



On the basis of the nature and inr"f.S.ve of problems found among proj- 
ects and their subsequent verificacicn vhrough other means, it is 
concluded that the 1974 SEPA was an effective means of problem detection 
among projects. 
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TABLE 3 



ANALYSIS OF PROJECT GROUP PERFORMANCE 
ON AUDIT INVENTORY SECTIONS 



STATUS REPORT 


PROJECT GROUPS BY FUNDING STATUS 


RATES BY 
EPA StCTIUNd 


0-1 -A 


0-1-B 
(N-9) 


0-2 

m) 


0-3 
(N-7) 


DM-1 
(N«3) 




SECTION 


I: PROJECT MANAGEMENT 




% + 
% NA 


76 

24 

0 


68 
30 
2 


50 
50 
0 


75 
25 
0 


6? 
33 
0 




SECTION 


II: PRODUCTION MANAGEMENT 




% + 
% - 
% NA 


52 
15 
33 


50 
35 
15 


38 
15 


52 
18 
30 


70 
23 
7 




SECTION 


III: MEASUREMENT OF CHANGE 




% + 
% NA 


70 
13 
17 


51 
42 

7 


62 
32 
6 


71 
21 
2 


36 
20 
44 




SECTION 


IV: PRODUCT DEVELOPMENT 

* 




•'. + 
i NA 


0 
0 

100 


52 
14 
34 


67 
33 
0 


67 
28 
5 


77 
23 
0 


SECTION V: PROGRAM OPERATION 


% + 
% - 
% NA 


100 
0 
0 


78 
22 
0 


81 
19 
0 


92 
4 
4 


100 
0 
0 


DESIGNATIONS - 


% + = Rate of Consistencies per Cell 

- - Rate of Discrepancies + Omissions per Cell 
" NA = Rate of Non -Applicable Items per Cell 
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The second purpose of the analysis in Table 3 was to locate high 
non-applicability rates within the five sections for projects in 
different operational stages. Such information indicated the lack 
of content validity for various sections of the audit inventory when 
used for projects in differing phases of operation. Two high non- 
applicability rates were found: 

(1) 0-1-A projects for Product Development, 100% non-applicable; and 

(2) DM-1 projects for Measurement of Change, 44% non-applicable. 



Section IV - Product Development was designed for project products which 
were completed and deployed for use. 0-1-A projects, having been oper- 
ational for only six weeks at the time of the audit, had no completed 
products to be inspected. The items designed for Section III - Measure- 
ment of Changes were designed for student performance evaluation. DM-1 
projects work with potential adopters of practices and products. The 
data targets for the evaluation of DM-1 projects were not congruent with 
the data targets for the evaluation of DM-1 projects were not congruent with 
parallel sections with alternate items appropriate for such projects 
must be developed for the 1975 audit inventory. 



Aggregate Audit Findings (Narrative Audit Items ) 

Section V of the audit Inventory dealt with Program Operation. Section 
V contained four narrative items to be prepared by the program specialist 
on the audit team. Two of these Items requested descriptions of the 
organizational point(s) at which intervention is provided to learners 
(e.g., an "intensive intervention program of high-risk learners" may be 
installed within a classroom, elementary level, "grade one). The other 
item relating to intervention (V-B-2) requested a brief description of 
the basic unit in which intervention occurs. (Basic unit descriptions 
should have contained resource, procedure, and time factors). 



An appraisal of the descriptions provided was performed using three 
judgmental criteria: completeness, clarity, and accuracy. The results 
of this appraisal are presented in Table 4. 
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TABLE 4 

APPRAISAL OF NARRATIVE DESCRIPTIONS REGARDING 
PROJECT INTERVENTION PRESENTED IN REPORTS 
(SECTION V-B) 



COMPLETE, CLEAR 
ACCURATE 


AMBIGUOUS. INCOMPLETE 
INNACURATE 


NUMBER 


1 PERCENTAGE 


NUMBER 


PERCENTAGE 


V-B-1: POINT{S) OF INTERVENTION 




1 36 




1 64 


V-B-2: BASIC UNIT(S) OF INTERVENTION 


17 


56 


13 
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As can be seen m Table 4, the descriptions provided were judged as 
complete, clear, and accurate in 36% of the reports for Item V-B-1 
and in 56r of the reports for Item V-B-2. ' • 

There were two narrative items in Section V-C of the audit Inventory. 
The program auditor was asked to provide narrative answers for opinion 
questions pertaining to the solution being developed by the project 
for the problem it was addressing. The questions were as follows: 

V-C-1: Do all program components interrelate with one another 
demonstrating both uniformity of fit and integrity as 
a whole: and 

V-C-2: To what extent does the program as a comprehensive 
solution afford coverage for the learner needs and 
problems addressed? 

An appraisal of the descriptions was performed using the same judgmental 
criteria: completeness, clarity, and accuracy. The results are presented 
in Table 5. 
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TABLE 5 

APPRAISAL OP NARRATIVE DESCRIPTIONS REGARDING 
PROJECT SOLUTION PRESENTED IN REPORTS 
(SECTION V-C) 



cHAftAc'T£fet2AtiohiS OF ktpm mmtm 



COMPLETE. CLEAR 
ACCURATE 


AMBIGUOUS, INCOMPLETE 
INNACURATE 


NUMBER 


PERCENTAGE 


NUMBER 


PERCENTAGE 


V-C-1: INTEGRITY OF SOLUTION COMPONENTS 


17 


56 


13 


44 


V-C-2: COMPREHENSIVENESS OF SOLUTION 


15 


50 


15 


50 



The appraisal revealed that the descriptions were judged as complete, 
clear, and accurate in 56% of the reports for Item V-C-1, and in 50* 
of the reports for Item V-C-2. 



Tables 4 and 5 show that about half of the auditor narrations prepared 
in response to the four items in Section V were characterized as 
ambiquous, incomplete, or inaccurate. This casts doubt on both the 
information provided in reports and the format used in Section V to 
request such information for reports. It is suspected that the format 
of the items in question did not define adequately the Information 
sought, or did not direct the auditor to report the information In 
an appropriate manner. 



To solve this problem, an alternative format will be used. Objective 
response items with a more narrow scope will be developed along with 
improved directions and illustrative examples. A review of the 
narrative description responses to the two questions in V-C was made. 
In the review, auditor responses were interpreted as either positive 
or negative to the questions asked by the Items. The results of this 
review are presented in Table 6 on page 42. • 
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TABLE 6 

INTERPRETED NARRATIVE RESPONSES REPORTED 
SECTION V-C (SOLUTION) 



lNTEftr>feEtATl6N 6^ NARfeAYtVE RgSpgRSIg 



AFFIRMATIVE RESPONSE 


NEGATIVE RESPONSE 


NUMBER 


PERCENTAGE 


NUMBER 


PERCENTAGE 


V-C-1: INTEGRITY OF SOLUTION COMrWENTS 


28 1 


t 92 1 


z 1 


I 8 


V-C-2: COMPREHENSIVENESS OF SOLUTION 


29 


96 


1 


4 



TI.e review and interpretation revealed that affirmative responses 
were given in 92% of the reports for Item V-C-1 (intearity of solution 
components) and in 96% of the reports for Item V-C-2 (comprehensive- 
ness of the solution). 



The audit report contained one additional narrative item. The item 
was to be jointly prepared by both members of the audit team. The 
item requested a summary opinion indicating the degree to which the 
project was being conducted in accordance with the terms and condi- 
tions of the grant application. 



A review of these opinion statements was performed. In the review 
process, the narrative statements were interpreted and sorted into 
three qeneral response categories: 

1. those indicating an affirmative compliance response; 

2. those giving an ambivalent compliance response; and 

3. those giving a negative compliarice response. 
The obtained results are presented in Table 7. 
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TABLE 7 



INTERPRETED NARRATIVE RESPONSES REPORTED 
FOR GRANT COMPLIANCE SUMMARY 



INtERPRfeYATION OF NARRATIVE RESPONSES 

■ ' I ' i ' 



AFFIRMATIVE 


AMBIVALENT 




NEGATIVE 




COMPLIANCE 


COMPLIANCE 




COMPLIANCE 


RESPONSE 


RESPONSE 




RESPONSE 




NUMBER 


Of 

li 


NUMBER 


% 


NUMBER 




24 


80 


5 


16 


1 


4 



The review and interpretation revealed that affirmative compliance 
responses were reported for 80% of the projects, that ambivalent 
responses were reported for 16% of the projects, and that a negative 
compliance response was given for one of the thirty projects audited. 



The findings reveal that in five instances audit teams failed to answer 
the grant compliance question with an affirmative or negative response. 
Such ambivalent responses possibly may be attributed to three causes-: 

1. an attempt by the audit team to evade the question, perhaps 
to avoid giving a negative response; 

?. dichotomous opinions held by the two members of the audit 
team which were not resolved; or 

3. imprecise narration by the writers. 

To solve the problem of ambivalent meaning in response to the opinion 
of grant compliance item, the response format will be modified for 
the 1975 SEPA Protocols. Boxes for affirmative or negative responses 
will accompany the item. Space will then be provided for a narrative 
justification for the response given by the auditors in the box. This 
modification should reduce the opportunity for evasion and should force 
imprecise writers to address the central point. Procedures also will 
be included for auditors to employ if the two members of the audit 
team should hold contrary opinions on the grant compliance issue. 
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Basis for Corrective Interventio n 

The audit findings presented herein have highlighted certain specific 
items and general areas in which discrepancies In project performance 
occurred. In the perceptions of project managers, auditors, and staff 
of the Educational Innovations Section such performance discrepancies 
are considered "problems." Some of the identified f'oblems potentially 
could prevent the completion or success of project missions. The test 
of such information is not its collection alone, but its application to 
a productive use. 



If the purpose of SEPA is the detection of problems, then the productive 
use of problem information is the design and delivery of corrective 
intervention. Both the project manager and the state agency have a role 
and responsibility to fulfill In the process of problem correction. The 
state agency must assist project managers in analyzing problems and 
exploring solution alternatives. The project managers then must select 
and Implement the best alternatives for their projects. The state 
agency through monitoring and the expert review and negotiation of 
plans, must determine whether the necessary changes have been Implemented. 
Both parties eventually must determine If the problem has been solved. 



The state agency has an additional responsibility to fulfill. Due 
to its unique purview over many projects, the state agency must design 
and deliver corrective intervention for common problems of major conse- 
quence and high incidence among projects. Traditional intervention 
approaches for such problems are workshops, training sessions, and 
modifications In guidelines. 



Between January 1, and July 30, 1974, training sessions for all projc-t 
managers, small group workshops, and technical assistance for individual 
projects were orovided. The review and negotiation of subsequent grant 
applications by the Educational Innovations Staff has shown that many 
audit detected problems have been addressed with corrective procedures. 



From this initial application of Intervention measures and a cursory 
examination of changes in projects, it appears that audit-produced 
Information can be applied effectively in treating detected problems. 
However, the best indication of the effectiveness of corrective 
intervention will be provided in the 1975 SEPA findings. A comparison 
of performance results from 1974 to 1975 on the same items applied 
to projects by the same or similar audit teams should indicate the 
extent to which detected problems have been solved. The rate at which 
problems are solved should provide evidence of the effectiveness of 
the corrective intervention provided. 
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Potentialities of the Standardized Educational Program Audit 

With the development of a standardized audit procedure which provides 
aqqreqate, objective, and cumulative information about the performance 
of projects, new and productive uses must be found for the information 
obtained. Perhaps procedures for preventive and corrective inter- 
vention can be developed and their effectiveness tested In future 
projects and selected problem-solving endeavors. One conclusion 
can be drawn at this point: the full range of productive data usage 
from the SEPA procedure now is unknown. Intuitively, it would 
seem that a standardized audit procedure with the capability of pro- 
viding problem detection information across a variety of educational 
activities at a given moment in time could have many potential appli- 
cations in the broad area of educational management. 
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